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Introduction 

SEI ITA  Background 

SEI  conducts  Independent  Technical  Assessments  (IT As)  on  large 
software-reliant  acquisition  programs 

•  ITAs  are  objective  program  reviews  of  people,  programmatics,  processes, 
technical  aspects,  and  the  environment 

•  ITA  teams  conduct  interviews  &  review  documents  on  program  status/history 

•  Identify  likely  causes  of  schedule,  cost,  or  performance  issues 

•  Recommend  improvement  or  recovery  actions 

SEI  brings  to  the  assessments 

•  Software,  systems  engineering  and  program  management  expertise 

•  Independent  and  neutral  third-party  assessment 

•  Experience  in  conducting  over  100  ITAs  and  Red  Teams 
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Introduction 

ITA  Pattern  Analysis  Objectives 

Identify  recurring  patterns,  both  positive  and  negative,  that  the  SEI  has 
observed  across  this  set  of  ITAs: 

•  Strengths 

•  Best  practices 

•  Weaknesses 

•  Issues 

Provide  practical  information  on  acquisition: 

•  Identify  underlying  causes  recurring  problems 

•  Make  actionable  recommendations  to  address  current,  and  to  prevent  future 
problems 
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Introduction 

Approach 

Gather  data  from  12  Air  Force  programs  reviewed  between  2006  and  2009: 

•  6  IT  system  programs 

•  2  Command  and  Control  (C2)  programs 

•  2  communications  system  programs 

•  1  avionics  system  program 

•  1  electronic  warfare  system  program 

Perform  qualitative  analysis  of  findings 

•  Divide  out  information  by  system  type  in  relevant  areas  (i.e.,  IT  systems) 

•  Consider  relevant  information  from  other  acquisition  programs 

Identify  higher-level  relationships  across  the  findings 

Identify  potential  root  causes  of  cost,  schedule,  scope,  and  quality  issues 

Recommend  corrective/preventative  strategies  based  on  these  patterns 
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Introduction 

Limitations 

ITA  data  is  inherently  qualitative 

•  Sample  set  of  12  programs  is  small 

•  Some  ITAs  were  focused  on  one  aspect,  such  as  testing 

•  Data  was  not  collected  with  intention  that  it  be  used  quantitatively 

•  Data  is  biased  by  different  ITA  team  expertise  areas 

•  Programs  were  selected  because  they  were  already  in  trouble 

The  most  frequent  findings  may  not  be  the  most  important  ones 

Fundamental  root  causes  may  not  be  explored  by  ITAs 

•  Root  causes  not  always  needed  to  make  practical  recommendations 

•  ITA  work  is  focused  on  helping  the  program — not  doing  research 

•  Example :  Untrue  that  “Poor  estimate”  means  “Can’t  do  good  estimates” 

Best  practices  may  not  always  be  found  by  ITAs 

•  Focus  is  primarily  on  identifying  issues  to  be  remedied 
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Findings 

Most  Common  Findings 

Inadequate  PMO  staff  expertise 
Hostility  between  stakeholders 

Poor  contractor  oversight  by  PMO  ( too  reliant  on  contractor) 
Insufficient  PMO  staff 
Poor  user/stakeholder  involvement 
High  PMO  staff  turnover 
Ineffective  risk  management 
Overly  optimistic  schedule 

Poor  contractor  oversight  by  PMO  ( insufficient  metrics) 
Requirements  scope  creep 

Inadequate  requirements 
Unpredictable  delivery  dates 
“Big  Bang”  integration 
Immature  technology 
Lack  of  functional  requirements  baseline 
Lack  of  Integrated  Master  Schedule  (IMS) 

Poor  process  adherence 
Unanticipated  technical  complexity 


9  occurrences 
8  occurrences 

6  occurrences 


5  occurrences 


4  occurrences 
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Findings 

Top  10  Overall  Categories  for  Findings 


Category 

Percent 

Aspects 

Staffing 

20% 

Expertise,  turnover,  staff  size 

Requirements 

10% 

Adequacy,  clarity,  creep,  baseline 

Oversight 

8% 

Adequacy,  metrics 

Schedule 

8% 

Master  schedule,  predictability 

Testing 

7% 

Fidelity,  adequacy,  hardware,  data 

Technical 

6% 

Complexity,  maturity 

Culture 

6% 

Inter-team  relationships 

Organizational 

5% 

Management,  formality,  dispersion 

Stakeholder  Involvement 

4% 

Level  of  involvement  with  program 

Risk  Management 

3% 

Effectiveness 
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Findings 

Key  IT  System  Findings 

Ineffective  User/Stakeholder  Involvement 

•  Stakeholders  not  adequately  involved  in  requirements  or  testing 

Poorly  Executed  Change  Management 

•  Little  account  for  system  impact  on  existing  business  processes 

•  Often  resulted  in  (avoidable)  user  resistance  to  the  new  system 

Lack  of  Program  Management  Rigor 

•  Business  (vs.  acquisition  or  IT)  people  were  running  the  program 

•  Requests  for  new  requirements  not  constrained — drove  cost/schedule 

•  Inappropriate  contractual  vehicles 

Technical  Complexity  is  Rarely  an  Issue 

•  Technical  complexity  was  not  a  significant  issue  for  most  IT  systems 
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Findings 

Continuing  and  Emerging  Trends 

Contracted  PMO  Staff 

•  This  ongoing  trend  will  be  reversed  by  plans  to  bolster  the  acquisition  workforce 

Interoperability  and  Open  Systems 

•  Leveraging  of  system  capabilities  through  interoperability  is  expected  to  grow, 
building  on  modular  design  and  open  standards,  moving  toward  SOA 

Joint/Common  Programs 

•  More  expected  to  help  reduce  costs,  despite  real  management  challenges 

Geographically  Distributed  Teams 

•  Continuing  growth  of  dispersed  teams  is  increasing  risk  of  poor  performance 

Internet/Web  Applications 

•  Need  for  Web  access  to  key  IT  systems  is  forcing  legacy  modernization  efforts 

Enterprise  Resource  Planning  (ERP) 

•  Increasing  ERP  use  for  IT  systems  driving  business  process  changes 

Agile  Development 

•  Some  interest  in  integrating  agile  methods  with  DoD  5000.02 
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Analysis 

Possible  Relationships  Among  Findings 

Program  Management  by  “Functionals” 

—►leads  to  low  PMO  staff  experience,  which... 

—►leads  to  overreliance  on  contractor,  which... 

—►leads  to  poor  contractor  oversight,  which... 
pleads  to  unpredictable  delivery  dates 

Geographically  Separated  Sites 

-^lead  to  poor  communication/cooperation,  which... 
pleads  to  conflict  across  sites 


Inadequate  PMO  Staff  Experience 

— ►  leads  to  poor  stakeholder  involvement,  which. . . 
pleads  to  inadequate  requirements,  which... 
—►leads  to  unplanned  rework,  which... 

—►leads  to  schedule  slip 


Need  to  ‘Sell’  the  Program 


pleads  to  overly  optimistic  schedule,  which... 
pleads  to  schedule  pressure,  which... 
pleads  to  contractor  sacrificing  quality  processes,  which... 
pleads  to  unplanned  rework,  which... 
pleads  to  schedule  slip 
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Analysis 

Candidate  Root  Causes  .1 


Geographically  Separated  Sites 

•  Separated  sites  have  extra  coordination  overhead  and  poor  visibility,  causing 
delays  and  frustration  that  may  turn  into  mutual  suspicion  and  growing  conflict 

Use  of  Advanced/Immature  Technology 

•  Users,  government,  and  contractors  all  prefer  highly  advanced  technology — but 
its  inherent  immaturity  drives  up  risk  and  cost,  and  lengthens  schedule 

Diminished  Acquisition  Workforce 

•  Inexperienced  PMO  staff  are  less  able  to  properly  select  and  oversee  technical 
contractors,  and  thus  less  able  to  ensure  successful  outcomes 

Ambitious  Requirements 

•  The  desires  for  higher  capability  and  “compelling”  programs  drive  ambitious, 
unprecedented  requirements  that  increase  complexity  and  risk 

Long  Program  Duration 

•  Large  programs  have  long  schedules — during  which  environment  changes  drive 
scope  changes,  causing  even  longer  schedules  and  higher  cost 
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Acquisition  Dynamics  Analysis 

Long  Program  Duration  -  “Longer  Begets  Bigger” 
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Analysis 

Candidate  Root  Causes  .2 

Instability  of  Program  Funding 

•  Political  concerns  produce  funding  volatility  that  consumes  effort  in  replanning, 
requiring  programs  to  extend  schedule  or  reduce  scope 

Military  Rotations 

•  Short-term  PM  rotations  place  emphasis  on  near-term  program  health,  creating 
incentives  to  put  off  longer-term  investments  that  have  no  immediate  benefits 

Underestimation 

•  Both  the  PMO  and  contractor  have  incentives  to  underestimate  cost  to  ensure 
that  a  program  is  funded — or  else  they’re  both  out  of  a  job 

Joint  Programs/Common  Infrastructure 

•  Common  infrastructure  programs  must  reconcile  competing  needs  into  one 
system — but  this  drives  up  cost  and  schedule,  and  drives  user  programs  away 
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Acquisition  Dynamics  Analysis 

Joint  Programs  -  “Everything  for  Everybody” 


o 


Amortized  Cost 
of  Shared 
Infrastructure 
per  Platform 


Platform 
Confidence  in 
Shared 
Infrastructure 


Number  of 
Platforms 
Interested  in 
Shared 
Infrastructure 


B 


Shared 

Infrastructure 

Complexity 


Time  to 
Deployment  of 
Shared 
Infrastructure 


Total  Cost  of 
Custom 
Capability 
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Analysis 

Mitigating  Root  Causes  .1 

Geographically  Separated  Sites 

•  Favor  the  use  of  co-located  developers  whenever  possible 

•  Substantially  invest  in  regular  on-site  presence  at  other  sites  through  travel 
with  face-to-face  contact  with  other  sites. 

Use  of  Advanced/Immature  Technology 

•  Increase  use  of  Technology  Readiness  Assessments  (TRAs)  to  improve 
visibility  of  the  technology  maturity 

•  Independently  review  PMO  choices  of  technologies  to  be  assessed 

Diminished  Acquisition  Workforce 

•  Improve  qualifications  of  acquisition  staff  emphasizing  software  expertise,  and 
improve  compensation  and  advancement  opportunities  to  increase  tenure. 

Long  Program  Duration 

•  Divide  large  acquisition  development  efforts  into  multiple  smaller,  shorter 
duration  programs. 

Instability  of  Program  Funding 

•  Buffer  programs  from  funding  variations  to  improve  stability  and  productivity. 
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Analysis 

Mitigating  Root  Causes  .2 

Military  Rotations 

•  Assign  PMs,  DPMs,  and  other  key  positions  for  the  program’s  duration  and 
into  deployment.  Use  civilians  if  military  rotations  are  not  amenable. 

Underestimation 

•  Don’t  require  PMO  to  adopt  contractor’s  estimate  for  the  program — or  else  use 
the  difference  as  PM  “reserve” 

•  Change  from  traditional  50%  estimation  confidence  level  to  80%  level 

•  DoD  should  consider  use  of  Vickrey  “second  price”  auction  mechanism  for 
acquisition  proposal  bidding 

Joint  Programs 

•  Consider  oversight  above  Senior  Acquisition  Executive  (SAE)  level  to  help 
ensure  cooperation  among  multi-Service  stakeholders. 
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Analysis 

Overarching  Themes 

It’s  the  People,  Not  the  Software 

•  Software  engineering  issues  are  rarely  the  main  reason  programs  fail 

•  Technical  issues  accounted  for  only  6%  of  the  ITA  findings 

The  Need  to  Sell  the  Program 

•  Acquisition  promotes  ‘selling’  programs  with  ‘unfounded  optimism  and 
parochialism’ 

The  Evolution  of  “Science  Projects” 

•  Prototypes  that  grow  in  scope  during  development  often  fail  the  transition  to 
become  production-quality  systems 

Common/Joint  Programs  Replace  “Islands  of  Automation” 

•  The  temptation  of  an  ideal  custom  solution  vs.  a  shared  “one-size-fits-all” 
system  is  often  too  great  for  stakeholders  to  resist 

Misaligned  Incentives 

•  People  are  too  often  incented  to  do  what’s  best  for  themselves,  at  the  expense 
of  their  organization  or  larger  community 
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Analysis 

Misaligned  Incentives 

The  acquisition  system  incentivizes... 

PMOs  to  ‘sell’  programs,  even  when  making  poor  progress 

PMOs  to  downplay  risks,  even  if  they  may  jeopardize  the  program 
PMOs  to  do  “big  bang”  integration  to  shorten  schedule,  despite  the  risk 
PMOs  to  choose  the  low  bidder,  even  if  it  may  cause  poor  performance/quality 

•  Contractors  to  underbid  programs,  and  then  overrun  cost/schedule 
Contractors  and  PMOs  to  use  immature  technology,  driving  up  cost/schedule 

•  Contractors  to  move  expert  staff  off  awarded  programs,  onto  proposed  programs 

•  Services  and  contractors  to  prefer  siloed  systems  over  Joint  programs 
Military  personnel  to  leave  programs  soon  after  they  become  valuable  staff 
Cost-Plus  contracts  that  inadvertently  encourage  longer  programs 

DoD  to  fund  too  many  programs,  thus  underfunding  all  of  them 
Users  to  demand  exotic  features,  because  they  bear  no  cost  for  doing  so 

.  ..and  these  behaviors  indirectly  drive  many  key  reasons  for  failure 
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Acquisition  Analysis  at  the  SEI 

For  Additional  Information 


Upcoming  SEI  Technical  Note:  “An  Analysis  of  Recurring  Issues  Found 
Across  12  U.S.  Air  Force  Software-Reliant  Acquisition  Programs ” 


Website:  http://www.sei.cmu.edu/acquisition/research/archetypes.cfm 
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“ Acquisition  Archetypes ”  analyze 
recurring  patterns  in  actual  programs, 
and  recommend  interventions  and 

preventative  actions: 

•  Firefighting 

•  Brooks'  Law 

•  "Happy  Path"  Testing 

•  Longer  Begets  Bigger 

•  The  Bow  Wave  Effect 

•  Shooting  the  Messenger 

•  Feeding  the  Sacred  Cow 

•  Everything  for  Everybody 

•  Underbidding  the  Contract 

•  Robbing  Peter  to  Pay  Paul 

•  Staff  Burnout  and  Turnover 

•  PMO  vs.  Contractor  Hostility 


gall  11  Software  Engineering  Institute 


Carnegie  Mellon 
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